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Using  a  Knowledge-based  Approach 
To  Improve  Weapon  Acquisitions 


_ ▲ _ 

This  booklet  describes  the 

KNOWLEDGE-BASED  APPROACH 
SUCCESSFUL  ORGANIZATIONS 
GENERALLY  FOLLOW  TO 
DELIVER  SOPHISTICATED 
PRODUCTS  IN  LESS  TIME  AND 
AT  LOWER  COSTS. 

- ▼ 


The  Department  of  Defense  (DOD)  is  on  the  threshold  of 
several  major  investments  in  acquisition  programs  that  are 
likely  to  dominate  budget  and  doctrinal  debates  well  into  the 
next  decade.  Over  the  next  5  years  alone,  DOD’s  overall 
investments  are  expected  to  average  $150  billion  a  year  as 
DOD  works  to  keep  legacy  systems  as  well  as  transform  our 
national  defense  capabilities  for  the  future.  To  meet  this 
challenge,  it  is  essential  that  sound  foundations  for 
investments  in  systems  be  laid  now  so  that  the  resulting 
programs  can  be  executed  within  estimates  of  available 
resources. 


At  the  request  of  the  Congress,  we  have  been  examining  ways  DOD  can 
optimize  its  investment  in  weapon  systems,  drawing  on  lessons  learned  from 
the  best,  mostly  commercial,  product  development  efforts.  Leading 
commercial  firms  we  have  studied  have  developed  increasingly  sophisticated 
products  in  less  time  and  at  lower  cost.  Key  to  their  success  is  their 
knowledge-based  approach  to  the  acquisition  of  new  products.  A  knowledge- 
based  approach  is  supported  by  incentives  that  encourage  realism  and  candor. 

This  booklet  highlights  the  results  of  our  work  to  date.  We  continue  to  explore 
additional  facets  of  the  acquisition  process  to  identify  best  practices.  More 
details  on  our  work  can  be  found  in  the  reports  cited  in  this  brochure. 

GAO’s  best  practices  work  is  done  under  the  direction  of  Katherine  V. 
Schinasi.  For  more  on  this  work,  go  to  www.gao.gov/bestpractices.  If  you  have 
questions  or  would  like  to  discuss  our  reviews,  please  contact  Ms.  Schinasi  at 
(202)  512-4841  or  schinasik@gao.gov. 


Jack  L.  Brock  Jr. 

Managing  Director 

Acquisition  and  Sourcing  Management 
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A  Knowledge-based  Approach  Puts  Acquisition 
Programs  in  a  Better  Position  to  Succeed 

Leading  commercial  firms  expect  that  their  managers  will  deliver  high  quality 
products  on  time  and  within  budgets.  Doing  otherwise  could  result  in  losing  a 
customer  in  the  short  term  and  losing  the  company  in  the  long  term.  Thus, 
these  firms  have  adopted  practices  that  put  their  individual  program  in  a  good 
position  to  succeed  in  meeting  these  expectations  on  individual  products. 
Collectively,  these  practices  ensure  that  a  high  level  of  knowledge  exists  about 
critical  facets  of  the  product  at  key  junctures  during  its  development  and  is 
used  to  deliver  capability  as  promised.  DOD  has  recognized  the  need  to  adopt 
ways  of  doing  business  that  enable  it  to  achieve  similar  results  for  its  weapons 
acquisitions. 

Our  reviews  have  shown  that  there  are  three  critical  junctures  at  which  firms 
must  have  knowledge  to  make  large  investment  decisions.  First,  before  a 
product  development  is  started,  a  match  must  be  made  between  the 
customers’  needs  and  the  available  resources — technical  and  engineering 
knowledge,  time,  and  funding.  Second,  a  product’s  design  must  demonstrate  its 
ability  to  meet  performance  requirements  and  be  stable  about  midway  through 
development.  Third,  the  developer  must  show  that  the  product  can  be 
manufactured  within  cost,  schedule,  and  quality  targets  and  is  demonstrated 
to  be  reliable  before  production  begins.  If  the  knowledge  attained  at  each 
juncture  does  not  confirm  the  business  case  on  which  the  acquisition  was 
originally  justified,  the  program  does  not  go  forward. 

In  applying  the  knowledge-based  approach,  the  most-leveraged  decision  point 
of  the  three  junctures  is  matching  the  customer’s  needs  with  the  developer’s 
resources.  This  initial  decision  sets  the  stage  for  the  eventual  outcome — 
desirable  or  problematic.  The  match  is  ultimately  achieved  in  every 
development  program,  but  in  successful  development  programs,  it  occurs 
before  product  development.  In  successful  programs,  negotiations  and  trade¬ 
offs  occur  before  product  development  is  started  to  ensure  that  a  match  exists 
between  customer  expectations  and  developer  resources.  Leading  firms  thus 
make  an  important  distinction  between  technology  development  and  product 
development.  Technologies  that  are  not  mature  continue  to  be  developed  in 
the  technology  base — they  are  not  included  in  a  product  development. 

With  achievable  requirements  and  commitment  of  sufficient  investment  to 
complete  the  development,  programs  are  better  able  to  deliver  products  at  cost 
and  on  schedule.  When  knowledge  lags,  a  number  of  risks  are  introduced  into 
the  acquisition  process  that  can  result  in  cost  overruns,  schedule  delays,  and 
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inconsistent  product  performance. 

An  approach  that  enables  organizations  to  achieve  a  match  between  needs  and 
resources  is  evolutionary  product  development.  Under  this  approach,  basic 
requirements  are  achieved  first,  with  additional  capabilities  planned  for  future 
generations  of  the  product.  Because  product  development  is  incremental, 
achieving  knowledge  is  more  manageable.  Commercial  companies  have  found 
that  trying  to  capture  the  knowledge  needed  to  stabilize  the  design  of  a 
product  with  considerable  new  technical  content  is  an  unwieldy  task — 
especially  if  the  goal  is  to  reduce  development  cycle  times  and  get  the  product 
to  the  marketplace  as  quickly  as  possible.  With  evolutionary  development,  a 
product’s  design  uses  only  components  and  subsystems  whose  reliability  has 
been  proven  through  past  use  or  testing. 


Knowledge  Points  at  a  Glance 


The  knowledge-based  process  followed  by  leading  organizations  is  highlighted 
and  further  discussed  below. 


Knowledge 
point  1 


Technology 

Development 


Knowledge  Knowledge 


t 

Program  launch 


■  Knowledge  point  1:  Resources  and  needs  match.  Knowledge  point  1 
occurs  when  a  sound  business  case  is  made  for  the  product — that  is,  a 
match  is  made  between  the  customer’s  requirements  and  the  product 
developer’s  available  resources  in  terms  of  knowledge,  time,  and  money.  To 
determine  their  available  resources,  successful  companies  rely  on  current 
and  valid  information  from  predecessor  programs,  new  technologies  that 
have  demonstrated  a  high  level  of  maturity,  system  engineering  data,  and 
experienced  people.  Successful  companies  also  communicate  extensively 
with  customers  to  match  their  wants  and  needs  with  the  firm’s  available 
resources  and  with  its  ability  to  manufacture  an  appropriate  product. 

■  Knowledge  point  2:  Product  design  is  stable.  Knowledge  point  2  occurs 
when  a  company  determines  that  a  product’s  design  is  stable — that  is,  it 
will  meet  customer  requirements  and  cost  and  schedule  targets.  A  best 
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practice  is  to  achieve  design  stability  at  the  product’s  critical  design  review, 
usually  held  midway  through  development.  In  DOD,  the  critical  design 
review  occurs  when  the  first  phase  of  product  development — product 
integration — has  been  completed  and  the  second  phase — product 
demonstration — is  about  to  begin.  Completion  of  at  least  90  percent  of 
engineering  drawings  at  the  critical  design  review  provides  tangible 
evidence  that  the  design  is  stable. 

■  Knowledge  point  3:  Production  processes  are  mature.  This  level  of 

knowledge  is  achieved  when  it  has  been  demonstrated  that  the  product  can 
be  manufactured  within  cost,  schedule,  and  quality  targets.  A  best  practice 
is  to  ensure  that  all  key  manufacturing  processes  are  in  statistical  control — 
that  is,  they  are  repeatable,  sustainable,  and  capable  of  consistently 
producing  parts  within  the  product’s  quality  tolerances  and  standards — at 
the  start  of  production.  It  is  important  that  the  product’s  reliability  be 
demonstrated  before  production  begins,  as  investments  can  increase 
significantly  if  defective  parts  need  to  be  repaired  or  reworked. 


Measuring  Success  at  Key  Knowledge  Points 

The  organizations  we  have  studied  apply  a  variety  of  tools  and  measures  to 
gauge  the  level  of  knowledge  that  they  have  attained  before  making  major 
investment  decisions. 

■  Measuring  technology  readiness 

Many  programs  now  use  an  analytical  tool — technology  readiness  levels 
(TRL) — that  can  assess  the  maturity  level  of  technology  as  well  as  the  risk 
that  maturity  poses  if  the  technology  is  included  in  a  product  development. 
The  experiences  of  the  DOD  and  commercial  technology  development 
cases  we  reviewed  indicate  that  demonstrating  a  high  level  of  maturity 
before  allowing  new  technologies  into  product  development  programs  puts 
those  programs  in  a  better  position  to  succeed.  Simply  put,  the  more 
mature  technology  is  at  the  start  of  the  program,  the  more  likely  the 
program  will  succeed  in  meeting  its  objectives. 

There  are  nine  TRLs,  each  denoting  a  level  of  demonstrated  performance, 
beginning  with  concept — the  lowest  level  of  readiness — to  application, 
where  the  technology  has  been  “flight  proven”  under  mission  conditions. 
The  higher  the  TRL,  the  smaller  the  gap  between  the  technology’s  maturity 
and  the  product’s  requirements,  and  the  lower  the  risk  of  including  the 
technology  in  the  product’s  development.  Our  best  practices  work  has 
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shown  that  TRL  7 — demonstration  of  a  technology  in  an  operational 
environment — is  the  level  of  technology  needed  to  minimize  risks  when 
launching  an  acquisition  program. 

Using  TRLs  to  assess  a  technology’s  maturity  helps  decisionmakers  make 
informed  choices  about  product  development.  If  a  technology  lacks 
maturity,  decisionmakers  can  choose  to  delay  product  development  until 
the  technology  has  matured  sufficiently  or  lower  the  product’s 
requirements  so  that  a  less  advanced  but  proven  technology  can  be  used. 
The  more  a  technology  has  been  proven,  the  more  likely  the  product  is  to 
meet  its  objectives  and  the  more  likely  the  company  can  minimize 
unexpected  problems  and  avoid  potential  schedule  delays  and  cost 
overruns. 

■  Measuring  design  readiness 

The  organizations  we  studied  also  understood  the  importance  of  having 
disciplined  design  reviews  and  getting  agreement  from  the  stakeholders 
that  the  product’s  design  had  been  demonstrated  to  meet  requirements 
before  beginning  initial  manufacturing.  Each  organization  had  a  design 
review  process  that  began  at  the  component  level,  continued  through  the 
subsystem  level,  and  culminated  with  a  critical  design  review  of  the 
integrated  system  to  determine  if  the  product  was  ready  to  progress  to  the 
next  phase  of  development. 

In  addition  to  design  engineers,  a  cross-functional  team  of  stakeholders  in 
the  process  included  key  suppliers,  manufacturing  representatives,  and 
service  and  maintenance  representatives.  From  past  experience,  leading 
organizations  have  discovered  that  cross-functional  teams  provide  a 
complete  perspective  of  the  product. 

At  critical  design  review,  the  leading  organizations  we  studied  generally 
require  that  at  least  90  percent  of  the  engineering  drawings  be  completed. 
They  consider  engineering  drawings  to  be  a  good  measure  of  the 
demonstrated  stability  of  a  product’s  design  because  the  drawings  represent 
the  language  used  by  engineers  to  communicate  to  the  manufacturers  the 
details  of  a  new  product  design — what  it  looks  like,  how  its  components 
interface,  how  it  functions,  how  to  build  it,  and  what  critical  materials  and 
processes  are  required  to  fabricate  and  test  it. 
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Other  Important  Tools  and  Practices 

Our  reports  have  identified  other  tools  and  practices  that  enable  successful 
implementation  of  the  approach.  These  include: 

■  Systems  engineering  for  identifying  gaps  between  requirements  and 
resources  so  that  they  can  be  reconciled  through  effective  trade-offs  before 
product  development. 

■  Employment  of  integrated  product  teams  to  bring  together  in  a  single 
organization  the  different  functions  needed  to  design  and  manufacture  a 
product,  such  as  engineering,  finance,  test  and  evaluation,  and 
manufacturing. 

■  Supplier  management  approaches  that  optimize  relationships  with  both 
contractors  and  subcontractors. 

■  Using  earned  value  management  techniques  to  track  a  projects  progress 
and  assess  its  ability  to  meet  cost  and  scheduling  goals. 

■  The  use  of  a  variety  of  testing  and  evaluation  tools  and  techniques  to 
validate  a  product’s  performance  early  and  throughout  development. 

■  The  use  of  targeted,  hands-on  methods  to  ensure  that  program  offices  are 
trained  on  key  practices. 

■  Collection  of  statistical  process  control  data  to  ensure  manufacturing 
processes  are  consistently  producing  parts  within  quality  standards. 

■  Considering  reasonable  operating  and  support  costs  and  the  readiness 
or  availability  of  equipment  as  requirements  equal  in  importance  to  other 
performance  characteristics. 

Coupling  the  knowledge-based  approach  with  sound  tools  and  techniques  for 
oversight,  strengthening  the  workforce,  and  ensuring  the  right  knowledge  is 
attained  at  the  right  times,  increases  an  organization’s  potential  to  meet  cost, 
scheduling,  and  performance  targets.  It  is  still  essential,  however,  that  the 
right  incentives  and  resources  be  in  place  to  encourage  program  managers  to 
employ  such  measures. 
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Following  a  Knowledge-based  Acquisition  Process  Is 
Critical  for  Weapon  Acquisitions 

The  majority  of  weapon  system  acquisitions  we  have  reviewed  over  the  past 
several  decades  experienced  problems  during  acquisition  that  drove  up  costs 
and  schedules  and  increased  technical  risks.  Many  programs  have  been 
restructured  by  DOD  in  the  face  of  delays  and  cost  growth;  a  few  have  been 
canceled.  We  have  found  that  these  problems  are  largely  rooted  in  the  failure 
to  match  customer’s  needs  with  the  developer’s  resources — technical 
knowledge,  timing,  and  funding- when  starting  product  development.  In  other 
words,  commitments  were  made  to  delivering  capability  without  knowing 
whether  technologies  being  pursued  could  really  work  as  intended.  Time  and 
costs  were  consistently  underestimated.  Problems  that  surfaced  early  cascaded 
throughout  development  and  magnified  the  risks  facing  the  program. 

_ ▲ _ 

When  Knowledge  Is  Not  Attained 
At  Key  Junctures 

■  Launching  an  acquisition  program  before 
requirements  and  available  resources  are 
matched  can  result  in  a  product  that  fails 
to  perform  as  expected,  costs  more,  or 
takes  longer  to  develop. 

■  Failure  to  ensure  design  stability  about 
halfway  through  product  development  can 
result  in  design  changes  that  are  more 
costly  to  correct  later  in  development  or 
after  the  product  is  fielded. 

■  Entering  production  before  manufacturing 
processes  are  under  control  can  result  in 
product  defects  that  require  additional 
resources  to  rework  or  scrap — a  costly  and 
inefficient  practice. 

- ▼ 

On  a  number  of  programs,  DOD  has  shown  that  it  can  manage  its  weapon 
system  acquisition  process  to  ensure  important  knowledge  about  technology 
and  requirements,  design,  and  manufacturing  is  captured  and  used  to  make 
informed  and  timely  decisions  before  committing  to  substantial  development 
and  production  investments.  Moreover,  over  the  past  few  years,  DOD  has 
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made  constructive  changes  to  its  acquisition  policy  to  embrace  best 
practices — especially  those  related  to  technology  maturity  and  separating 
technology  development  from  product  development.  If  faithfully  implemented, 
DOD’s  policies  of  evolutionary  acquisition  and  phased  requirements  will  make 
it  easier  to  match  resources  with  needs  before  starting  a  new  product 
development.  Funding  pressures  and  the  need  to  modernize  systems  across 
the  Department  will  continue  to  make  it  paramount  for  DOD  to  adopt 
practices  that  can  save  money  and  deliver  new  capabilities  quicker. 

However,  a  number  of  incentives — many  tied  to  funding — continue  to 
undermine  DOD’s  ability  to  achieve  a  match  between  needs  and  resources 
at  the  onset  of  weapon  acquisition  programs.  Unlike  the  commercial  world 
where  the  focus  is  on  delivering  a  product  to  market,  DOD’s  system  focuses 
on  competing  for  resources.  In  the  competition  for  funding,  managers  are 
encouraged  to  launch  product  developments  before  technologies  are 
mature.  Because  a  proven  way  to  win  support  for  a  new  weapons 
acquisition  program  is  to  promote  unprecedented  performance  features 
and  design  characteristics,  managers  have  implicit  incentives  to  rely  on 
immature  technologies.  Moreover,  because  funding  is  competitive  and 
DOD’s  forecasts  of  cost,  schedule,  and  performance  are  largely  based  on 
immature  technologies  and  other  unknowns,  estimates  tend  to  be  squeezed 
into  insufficient  profiles  of  available  funding.  Other  factors,  such  as  short 
tenures  and  career  pressures,  discourage  program  managers  from  saying  no 
to  requirements  that  are  later  discovered  to  have  been  unreasonable.  Thus, 
to  meet  the  investment  challenge  of  modernizing  its  forces,  DOD  will  need 
not  only  to  implement  policies  that  embrace  evolutionary,  knowledge- 
based  acquisitions,  it  will  need  to  instill  incentives  that  encourage  realism 
and  candor  in  the  acquisition  process  and  sustain  its  commitment  to 
improving  business  practices. 
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Best  Practices  Reports 

These  reports  can  be  found  on  the  GAO  Web  site  at  www.gao.gov. 

Defense  Acquisitions:  Assessments  of  Major  Weapon  Programs,  GAO-03-476, 
Washington,  D.C.:  May  2003. 

Best  Practices:  Setting  Requirements  Differently  Could  Reduce  Weapon 
Systems’  Total  Ownership  Costs.  GAO-03-57.  Washington,  D.C.:  February  11, 
2003. 

Best  Practices:  Capturing  Design  and  Manufacturing  Knowledge  Early 
Improves  Acquisition  Outcomes.  GAO-02-701.  Washington,  D.G.:  July  15,  2002. 

Defense  Acquisitions:  DOD  Faces  Challenges  in  Implementing  Best  Practices. 
GAO-02-469T.  Washington,  D.C.:  February  27,  2002. 

Best  Practices:  DOD  Teaming  Practices  Not  Achieving  Potential  Results.  GAO- 
01-510,  Washington,  D.C.:  April  2001. 

Best  Practices:  Better  Matching  of  Needs  and  Resources  Will  Lead  to  Better 
Weapon  System  Outcomes.  GAO-Ol-288.  Washington,  D.C.:  March  8,  2001. 

Best  Practices:  A  More  Constructive  Test  Approach  Is  Key  to  Better  Weapon 
System  Outcomes.  GAO/NSLAD-OO-199.  Washington,  D.C.:  July  31,  2000. 

Defense  Acquisitions:  Employing  Best  Practices  Can  Shape  Better  Weapon 
System  Decisions.  GAO/T-NSLAD-OO-137.  Washington,  D.G.:  April  26,  2000. 

Best  Practices:  DOD  Training  Can  Do  More  to  Help  Weapon  System  Programs 
Implement  Best  Practices.  GAO/NSIAD-99-206.  Washington,  D.G.:  August  16, 
1999. 

Best  Practices:  Better  Management  of  Technology  Development  Can  Improve 
Weapon  System  Outcomes.  GAO/NSLAD-99-162.  Washington,  D.G.:  July  30, 
1999. 

Best  Practices:  DOD  Can  Help  Suppliers  Contribute  More  to  Weapon  System 
Programs.  GAO/NSIAD-98-87.  Washington,  D.G.:  March  17,  1998. 

Best  Practices:  Successful  Application  to  Weapon  Acquisition  Requires 
Changes  in  DOD’s  Environment.  GAO/NSIAD-98-56.  Washington,  D.C.: 
February  24,  1998. 

Best  Practices:  Commercial  Quality  Assurance  Practices  Offer  Improvements 
for  DOD.  GAO/NSIAD-96-162.  Washington,  D.G.:  August  26,  1996. 
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